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Is it now possible 
to articulate a 
general set of goals 
for student learning 
in STEM fields? 


Are NEW APPROACHES to transforming under- 
graduate learning in science, technology, 
engineering, and mathematics (STEM) making 
a difference? If so, how? How do we know? And 
what next? These are the questions explored in 
a 1999 report from Project Kaleidoscope, which 
concluded by making predictions and recom- 
mendations for the coming decade (Rothman 
and Narum 1999). Now that that “coming 
decade” is here, it is 
timely to ask how ac- 
curate those predictions were and to offer some 
new recommendations for the next decade. 

The predictions made in 1999 addressed a 
broad range of issues, from faculty to facilities 
and more. In each of the scenarios for the fu- 
ture that were developed then, the underlying 
theme was that attention to learning and as- 
sessment would be pervasive in the undergrad- 
uate STEM learning environment on campuses 
across the country. One reason we thought 
this would be the case had to do with the an- 
ticipated impact of How People Learn: Brain, 
Mind, Experience and School, a seminal report 
published that year by the National Research 
Council. The report called for the development 
of academic cultures where deep understanding 
about how students learn determines how 
courses and curricula are planned, technologies 
selected, spaces designed, and faculty recog- 
nized and rewarded. Further, it was a report that 
could be used as a resource for shaping and sus- 
taining such cultures. 

There were several other compelling reasons 
for basing our future scenarios upon the ex- 
pected emergence a new kind of learning cul- 
ture. In 1999, there was growing external 
pressure — from public agencies, accrediting 
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agencies, funding agencies, and the business 
community — for greater transparency with re- 
gard to student learning outcomes. New ac- 
creditation practices for engineering education 
programs were challenging that community of 
professionals, and many other STEM communi- 
ties were giving new or renewed attention to 
student learning outcomes in their specific dis- 
ciplines. Moreover, the National Survey of 
Student Engagement was piloted in 1999. 

Equally important was the increasing visi- 
bility and maturity of the work of pedagogical 
pioneers — agents of change whose efforts had 
been supported by the National Science Foun- 
dation (NSF) since the late 1980s. Their expe- 
riences and expertise were beginning to inform 
a generation of what dissemination literature 
calls “early adapters.” There was a growing body 
of research-based theory and practice about 
what works in the iterative cycle of exploring, 
examining, addressing, and assessing under- 
graduate student learning goals. 

So, where are we today? Where and when 
are conversations about students and student 
learning taking place within institutions or 
scholarly communities? How widespread among 
STEM faculty and their administrative colleagues 
is awareness of the work of pedagogical pioneers 
and of the growing body of research on learn- 
ing and cognitive science? Is it now possible to 
articulate a general set of goals for student 
learning in STEM fields on which local efforts 
can be built and against which they can be 
compared? And if so, what recommendations 
can be made for the next decade? 

Current conversations covering the range 
of issues related to student learning are dra- 
matically different from those of a decade ago. 
There is a growing national consensus about 
what students should know and be able to do 
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as an outcome of their careers 
as undergraduate learners. 

The work of the Association 
of American Colleges and 
Universities (AAC&U) — 
most importantly through its 
Liberal Education and Amer- 
ica’s Promise initiative — has been a significant 
catalyst in engaging communities in discus- 
sions about the kinds of learning needed for a 
complex and volatile world. Explicit and re- 
markably consistent goals for student learning 
have been articulated and promulgated by 
greatly diverse groups both within and beyond 
the academy. 

The Council on Competitiveness (2005, 76), 
for example, has called for preparing “a whole 
generation with the capacities for creative 
thinking and for thriving in a competitive cul- 
ture, able to work in multidisciplinary teams, . . . 


be comfortable with ambiguity, 
recognize new patterns within 
disparate data, . . . [and] to 
be inquisitive and analytical.” 
The undergraduate neuro- 
science community has out- 
lined learning goals, including 
critical thinking and independent thought; 
communicating effectively in written and oral 
form as well as with figures, graphs, and through 
presentation software; and an appreciation of 
the value of diversity and the ability to work 
with colleagues from a variety of backgrounds 
and perspectives (see Wiertelak 2003). The 
American Chemical Society’s Committee on 
Professional Training has stated that the 
outcome of laboratory experiences should 
“give students hands-on experience with 
chemistry and the self-confidence and compe- 
tence to . . . interpret experimental results and 


Attention to 
how people learn 
is beginning to 
transform the 
undergraduate STEM 
learning environment 
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draw reasonable conclusions, analyze data sta- 
tistically and assess reliability of results . . . and 
communicate eifectively in small groups and 
teams. . . ” (2003, 10). 

Are the conversations taking place within 
national groups changing the dialogue on 
campuses? One indication that they are is the 
fact that colleges and universities across the 
country are beginning to make public their ex- 
plicit visions of student learning by publishing 
them on their Web sites. The public announce- 
ment of the specific learning outcomes estab- 
lished by Miami Dade College is one of the 
most recent and most visible examples of the 
mainstreaming of attention to setting and as- 
sessing student learning goals both within and 
beyond STEM fields (see the article by Eduardo 
J. Padron in this issue of Liberal Education ). 

This brings me to the first of my recommen- 
dations for the coming decade, namely that 


leadership teams on campuses gather and distill 
lists of learning outcomes articulated by leader- 
ship associations such as AAC&U, as well as by 
other professional and disciplinary societies and 
corporate leaders, and translate those statements 
into a coherent and institutionally-appropriate 
set of goals for student learning that serves 
their vision for the future. Make those goals, 
and the actions that advance them, public 
and transparent. 

Pedagogies of engagement 

The most compelling evidence of the main- 
streaming of conversations about how people 
learn comes from the field. Just-in-Time Teach- 
ing, Problem-Based Learning, and Student- 
Centered Activities for Large Enrollment 
Undergraduate Programs are three examples 
of what Russell Edgerton (2001) has described 
as “pedagogies of engagement,” and they 
demonstrate the ways in which attention to 
how people learn is beginning to transform the 
undergraduate STEM learning environment. 

Just-in-Time Teaching (JiTT) involves dia- 
logue between student and student as well as 
between student and instructor, much of which 
occurs outside of the classroom — thanks, in 
part, to the maturation of electronic technolo- 
gies. Gregor Novak, one of the JiTT pioneers, 
says that at the heart of the JiTT pedagogy are 
pre-instruction, Web-based assignments called 
“warm-ups.” These are short, thought-provoking 
questions that, when fully discussed, often have 
complex answers. Students are expected to de- 
velop the answers as far as they can on their 
own, and then the job is finished by working 
together in the classroom. These warm-ups 
are submitted electronically just a few hours 
before class, giving the instructor (just) 
enough time to incorporate into the upcoming 
lesson the insights gained from student sub- 
missions. Exactly how the classroom time is 
spent depends upon a variety of issues such as 
class size, classroom facilities, and student and 
instructor personalities. 

In a JiTT classroom, students construct the 
same knowledge as in a passive lecture, but 
with two important added benefits. First, having 
completed the Web assignment very recently, 
they enter the classroom ready to engage ac- 
tively in their learning. Second, they have a 
feeling of ownership of their learning because 
the interactive lesson is based on their own 
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wording and understanding of the relevant is- 
sues. “Our goal,” Novak explains (pers. comm.), 
“is to create and sustain team spirit. We all, 
students and faculty, work together toward the 
same objective, that students pass the course 
with the maximum amount of enduring 
knowledge, skills, and habits of the mind that 
are critical for success in STEM communities 
of learners and practitioners.” Does JiTT 
work? Consider the following testimony from 
a student: 

It is easy to feel disconnected from a science 
course as a student. Each day can seem as a 
new set of notes to take from the instructor’s 
monologue, another chapter to read, and 
another problem set to work on, but each 
unrelated to the previous day — that is, until 
the exam. The situation changes if the as- 
signments are designed to pose questions 
that require some real effort and interaction 
with other students ahead of class, but pro- 
viding the assurance that the toughest points 
will be cleared up in the class makes that 
work worthwhile. Just-in-Time Teaching 
offers the kind of day-to-day motivation that 
drives the course forward for me. (Project 
Kaleidoscope 2007, 2) 

Problem-Based Learning (PBL) simulates 
workplace projects that require mastery of a 
range of content knowledge as well as the de- 
velopment and application of process skills in 
an integrative and interesting format. Faculty 
Albion College of the biomedical engineering program at 



Georgia Institute of Technology, for example, 
arrived at PBL as the foundation for planning a 
program from scratch. Theirs was a discipline 
without a history of pedagogies or a tradition 
of textbooks. (Admittedly, this can be seen as 
a luxury when trying to incorporate research 
on learning into the process of curricular 
change!) The inherent interdisciplinarity of 
biomedical engineering drew them into re- 
search on cognitive flexibility — that is, the 
ability to look at problems from a variety of 
perspectives. Their program was designed to 
challenge students with the right kind of 
problems, and the goal was to produce integra- 
tive problem-solvers who have the cognitive 
flexibility to apply engineering analysis and 
synthesis to problems in the biosciences. 

On the impact on student learning, Wendy 
Newstetter, one of the founding members of 
the PBL team at Georgia Tech, reports that 
solving problems on the frontiers of science 
that other experts are trying to solve at the 
same time does two things: it motivates 
students tremendously, and has a very inter- 
esting impact on identity. A major problem 
with students going into the sciences and 
being sustained is that they don’t identify 
with the kind of activity they are being asked 
to do. They don’t see their own personal 
identities or lives aligned with science. 
Whereas, when you give them complicated, 
multi-dimensional, interdisciplinary prob- 
lems from the real world, their imagination 
is sparked. They begin to say, “I can see 
myself doing this.” So problem solving is 
about motivation and identity, about en- 
gaging students through the excitement 
and fantasy of trying to solve those problems. 
(Project Kaleidoscope 2006) 
Student-Centered Activities for Large En- 
rollment Undergraduate Programs (SCALE-UP) 
is based on the conviction that, in order to un- 
derstand the science, students must actually do 
the science as a central activity in studying 
the science. SCALE-UP overcomes the many 
barriers to “doing” science in the traditional 
large lecture class: the isolation of individual 
students in the crowd of strangers, the compet- 
itive atmosphere, and the little one-on-one 
contact with the instructor. SCALE-UP uses 
cooperative learning pedagogical techniques 
with classes of approximately one hundred 
students, with lecture and lab integrated in 
technology-rich settings. Active group learning 
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is promoted within consistent 
groups, and the grading sys- 
tem requires teamwork to en- 
sure that each student in the 
group — even the really bright 
ones — benefits from working 
together. 

SCALE-UP pioneer Robert 
Beichner reports that substan- 
tive evaluation — video and 
audio recordings, interviews, focus groups, pre- 
and post-tests, and student profiles — has re- 
vealed improvements in students’ ability to 
solve problems and their attitudes toward 
science and learning, as well as increases stu- 
dents’ conceptual understanding. And SCALE- 
UP reduces failure rates. When asked what 
impression a visitor would get from walking 
into a SCALE-UP classroom, Beichner says, 
their impression would be that the learning 
space looks more like a restaurant than a 
classroom, or perhaps more like a banquet 
hall, because there is much noise from the 
visibly engaged students. They would see 
the realization of our idea that social inter- 
actions between students and their teachers 
is the “active” ingredient that works for us. 
From our own experiences and from re- 
search on learning, we knew that as stu- 
dents collaborate on interesting tasks they 
become deeply and personally involved 
with what they are learning. The doors of 
the closets and the walls of the classroom 
are covered with whiteboards — public 
thinking spaces — to help them share their 
learning with each other and their instruc- 
tor. (Project Kaleidoscope 2005) 

What we find in examining the core princi- 
ples of these and other pedagogies of engage- 
ment is that each of them can be seen, in some 
way, as a grandchild or great-grandchild of un- 
dergraduate research, which is surely the epit- 
ome of a pedagogy of engagement. Each of 
these contemporary pedagogies is designed to 
introduce students to the community of prac- 
tice in STEM fields and to enhance their per- 
sonal understanding of how scientists, 
engineers, and mathematicians make sense of 
our world. In other words, each invites stu- 
dents to assume the identity of the STEM pro- 
fessional. Each seeks to give students 
confidence in their ability to pursue study in 
STEM fields, based on successive (and presum- 
ably successful) social interactions within a 


collaborating community. 
These are communities in 
which expert and novice 
learners come together in the 
same spaces — sometimes 
physical, sometimes virtual — 
to engage in the process of dis- 
covery as they move from 
what is known to what might 
be known. These are pedago- 
gies designed to help students understand the 
importance of being a part of a collaborating 
community with a sense of shared purpose. 

Another similarity is that, although most 
began with support from the National Science 
Foundation, these pedagogies work for all disci- 
plines, serve all institutional types, strengthen 
the learning of all students, and reflect societal 
and disciplinary goals for undergraduate learn- 
ing. They all call upon instructors to reflect 
deeply on how to get students to take owner- 
ship of their learning, how to transform the 
learning environment from one of passive 
transmission of information to active construc- 
tion of knowledge and skills, and how to moni- 
tor the progress of their students’ learning. 

Barriers 

Despite the recognized national need for the 
kind of problem-solving, critical-thinking 
risk-takers formed through these pedagogies 
of engagement, their adoption is not yet 
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Project Kaleidoscope (PKAL) is a leading 
advocate for what works in building and 
sustaining strong undergraduate programs in 
the fields of science, technology, engineering, 
and mathematics (STEM). An informal 
alliance, PKAL takes responsibility for shaping 
undergraduate STEM learning environments 
that attract students to STEM fields and inspire 
them to persist and succeed. Such environments 
give students personal experience with the joy of 
discovery and an awareness of the influence of 
science and technology in the world. Resources 
derived from the work of the extensive 
PKAL community are available for adaptation 
by leaders on campuses across the country. 

Visit www.pkal.org for more information. 


Each of these 
contemporary 
pedagogies is 
designed to 
introduce students 
to the community 
of practice 
in STEM fields 
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widespread. One significant barrier to the 
spread of contemporary pedagogies of engage- 
ment is institutional culture. The compelling 
research conducted by Charles Henderson 
and Melissa H. Dancy (2007, 1) examines 
why “proven strategies are slow to integrate 
into mainstream instruction” even though, as 
they document, STEM faculty understand 
some of the problems with traditional ways of 
teaching. Henderson and Dancy conclude 
that situational characteristics consistent with 
traditional instruction account for the major 
impediments. Their suggestion, directed at 
the broader STEM research community, is 
really for us all: identify both the situational 
barriers faculty face — bolted-down chairs, 
large enrollment classes, the need to “cover” 
content, student lack of experience with 
active learning, etc. — and the means to re- 
moving those barriers. As they say, “after all, 
getting the chairs unbolted is often a non- 
trivial task.” 

This brings me to the second of my recom- 
mendations for the coming decade, namely 
that leadership teams gather critical informa- 
tion about interest and expertise relative to 
pedagogies of engagement within their cam- 
pus community and about local barriers to 
promoting and adapting those approaches; 
design and implement an action plan to over- 
come barriers in the process of adapting peda- 
gogical innovations to serve learning goals for 
the students for whom they are responsible. 


An oft-cited barrier to the mainstreaming 
of pedagogies of engagement is the lack of 
evidence of their efficacy. Over the past decade, 
however, this barrier has been largely sur- 
mounted by building on efforts to link research 
on STEM learning to STEM teaching — efforts 
that go back many years, at least to the work 
at the University of Washington by Arnold 
Arons and Lillian McDermott (see Narum and 
Rothman 1999). Although these efforts were 
present and becoming visible in 1999, the past 
ten years have seen a significant increase in 
efforts like those of Nobel Laureate Carl 
Weiman (2007) to take a scientific approach 
to pedagogical and institutional transformation. 

Among those leading these efforts are a cadre 
of assessment pioneers, who, with support from 
the NSF’s Assessing Student Achievement 
(ASA) program, have been deeply involved in 
research-based initiatives designed to get inside 
the learning process and gather evidence to 
document what works and for which students. 
Some start from what disciplinary content stu- 
dents should know (the Calculus Concept In- 
ventory, the Geoscience Concept Inventory, 
and Measuring What Students Know about 
How to Learn Chemistry). Others approach 
assessment directly from what skills students 
should acquire (Assessing Problem-Solving 
Strategies in Chemistry and Assessing Critical 
Thinking Skills) or from the STEM literacy 
perspective (Assessing Students’ Value for 
Science and Math Literacy). 

At a meeting of the ASA community in 
2006, project principal investigators were 
invited to share insights from their experi- 
ences with colleagues across the country. They 
urged STEM faculty to recognize that students 
know and understand less when they emerge 
from courses than most faculty think they do; 
that what we teach, despite our best efforts, is 
not what students learn or how they learn; 
that student achievement can be increased 
with effective assessment; and that you can 
teach better and enjoy it more if your students 
are demonstrably learning better. This group 
of experienced practitioners also thought that 
the situational barriers that keep others from 
exploring, adapting, and extending their work 
could be overcome if three conditions were 
met. First, STEM faculty and their administra- 
tive colleagues would need to realize there is 
no need to reinvent the assessment wheel — an 
effort costly in both time and energy. Second, 
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faculty would need to identify themselves as 
members of the community of STEM pedagogi- 
cal and assessment practitioners in the same 
way they feel an identity as a member of a 
STEM research community. And third, formal 
opportunities would need to be available at 
the local level — campuswide, within depart- 
ments or programs — for conversations about 
how difficult it is to teach certain concepts or 
students with different learning styles, conver- 
sations that capitalize on the expertise and in- 
terest resident in their community by engaging 
that broader community in asking, what works 
for our students, and how do we know? 

I hesitate to predict what the STEM world 
will look like in 2019, especially given how 
different it is today from the way the 1999 Pro- 
ject Kaleidoscope report predicted it would be. 
And so I will end instead with the third of my 
recommendations for the coming decade, 
namely that those with a stake in a robust 
twenty-first-century undergraduate STEM 
learning environment step back, take time to 
work with the communities of which they are 
a part to agree on outcomes from undergradu- 
ate engagement in STEM learning, determine 
their individual and collective responsibility 
for ensuring their students achieve those goals, 
and make it happen. □ 

To respond to this article, e-mail liberaled@aacu.org, 
with the author’s name on the subject line. 
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the United 
States must 
make science 
achievement a 
top priority. We 
need more suc- 
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students' engagement and achievement in science? 
This conference will focus on how colleges and 
universities — working together — can create a 
climate for engaging science for far-reaching 
educational change. 


For more information or to register: 
www.aacu.org | 202-387-3760 | network@aacu.org 


Project Kaleidoscope. 2005. Student-centered activities 
for large enrollment undergraduate programs 
(SCALE-UP): An interview with Robert Beichner. 
www.pkal.org/documents/Beichner_SCALEUP_ 
Interview.cfm. 

. 2006. Problem-based learning at Georgia 

Tech: An interview with Wendy Newstetter. 
www.pkal.org/documents/PBL_GeorgiaTech.cfm. 

. 2007. Just-in-time teaching. What Works: 

Reports from the Community 4: 2. www.pkal.org/ 

documents/All%20Pedagogies%20in%20one%20d 

ocument%20for%20posting.pdf. 

Rothman, F. G., and J. L. Narum. 1999. Then, now 
and in the next decade: A commentary on strengthening 
undergraduate science , mathematics , engineering and 
technology education. Washington, DC: Project 
Kaleidoscope. 

Weiman, C. 2007. Why not try a scientific approach 
to science education? Change 39 (5): 9-15. 

Wiertelak, E. 2003. Neuroscience education: Goals 
for the undergraduate program, www.pkal.org/ 
documents/GoalsUndergradNeuroscience.cfm. 


WSm 


o 


0. 

o 


o 

ui 

tc 

3 


< 

111 


Spring 2008 Liberal Education 19 





